fix(mcp): reduce describe_source timeouts by using rollup tables for map key discovery - #2426

🦋 Changeset detected

Latest commit: b4d6f7a

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jun 8, 2026 7:37pm
hyperdx-storybook	Ready	Preview, Comment	Jun 8, 2026 7:37pm

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

• Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

E2E Test Results

✅ All tests passed • 197 passed • 3 skipped • 1353s

Tests ran across 4 shards in parallel.

View full report →

Greptile Summary

This PR fixes clickstack_describe_source timeouts by hoisting metadataMVs and dateRange above Stage 2 so getMapKeys can use the fast rollup-table path for map attribute key discovery instead of always falling back to a full main-table scan. It also threads AbortSignal into getMapKeys and the per-key fallback queries in getMapValues so in-flight ClickHouse queries are cancelled when the 10s timeout fires.

• Rollup path enabled for Stage 2: metadataMVs and dateRange are now computed once before Stage 1 and passed down to getMapKeys in Stage 2, giving the rollup branch the data it needs to skip expensive main-table scans.
• Signal propagation: AbortSignal is wired into both the rollup ClickHouse query and the main-table fallback query inside getMapKeys, and into the per-key getMapValues calls in getAllKeyValues, so the ClickHouse server stops executing these queries as soon as the caller times out.

Confidence Score: 4/5

Safe to merge for the common case, but a timeout during the newly-enabled rollup query in Stage 2 will cache an empty key list for the entire aligned time window, causing all subsequent describe calls to silently return no map attribute keys until the process restarts.

The rollup-path closure in getMapKeys catches all errors — including the AbortError raised when signal fires — and returns []. getOrFetch stores this empty array in the process-level MetadataCache. Because the cache check runs before the rollup branch, every follow-on call for the same source/column/time-window hits the early-return and gets [], bypassing both the rollup and the main-table fallback. Before this PR, Stage 2 never reached the rollup path, so this latent behaviour was harmless; the new wiring makes it reachable for the first time.

The rollup try/catch block in packages/common-utils/src/core/metadata.ts (getMapKeys, ~line 428) needs attention — it should distinguish abort errors from genuine query failures before returning [].

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant C as MCP Caller
    participant DS as describeSource
    participant GK as getMapKeys
    participant RU as Rollup Table
    participant MT as Main Table

    C->>DS: clickstack_describe_source
    DS->>DS: compute metadataMVs + dateRange (hoisted above Stage 2)
    DS->>DS: Stage 1 - getColumns()
    DS->>GK: getMapKeys with metadataMVs + dateRange + signal
    note over GK: Check persistent MetadataCache
    alt rollup path available
        GK->>RU: SELECT Key FROM keyRollupTable WHERE ColumnIdentifier AND Timestamp range
        RU-->>GK: keys
        GK-->>DS: keys
    else rollup empty or unavailable
        GK->>MT: full table scan
        MT-->>GK: keys
        GK-->>DS: keys
    end
    DS->>DS: Stage 3 - getAllKeyValues LC columns
    DS->>DS: Stage 4 - getAllKeyValues map attribute values
    DS-->>C: schema + mapAttributeKeys + values

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into claude/fix-desc..." | Re-trigger Greptile}